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Abstract 

In this work, we propose a fast algorithm for multiple kernel learning (MKL). Our proposed approach 
builds on the original QCQP formulation of Lanckriet et al. Ifl2l . It uses a multiplicative weight update 
based approximation for the underlying SDP, exploiting a careful reformulation of the MKL problem as 
well as a novel fast matrix exponentiation routine for QCQP constraints that might be of independent 
interest. Our method avoids the use of commercial nonlinear solvers, and scales efficiently to much 
larger data sets than most prior methods can handle. Empirical evaluation on eleven datasets shows 
that our method is significantly faster than most current MKL methods and compares favorably with a 
uniform unweighted combination of kernels. 

1 Introduction 

Kernel methods have been extremely successful in machine learning applications [18, 19]. Since the success 
of these methods rely on an appropriate choice of kernel (or kernels), there has been extensive research on 
learning a combination of multiple kernels for a given task. This approach of learning multiple kernels 
outperforms lfL?l |2Q| algorithms that operate with a single kernel and, as a result, multiple kernel learning 
(MKL) has found successful applications in a wide variety of learning tasks and domains Ifl2l |4j |2l |27j |9j 
M MM- 

Support vector machines (SVM) use kernels directly, and a natural approach to multiple kernel learning 
is to jointly optimize the SVM task and the choice of kernels. This approach, pioneered by Lanckriet et al. 
Ifl2l . exploits convex optimization at the heart of both problems. It is theoretically elegant, but requires 
repeated invocations of semidefinite solvers which make the algorithm accurate but slow. Followup work 
focuses on two lines of attack: methods that speed up the solvers by replacing the semidefinite formulation 
with other optimizations, and methods that bypass the solvers by alternately optimizing the classifier and 
the kernel combination parameters. The latter approach, while faster, loses the theoretical guarantees of the 
former. Most importantly, none of these approaches scale effectively beyond a few thousand points. 

In this work, we present a theoretically sound and practically efficient MKL algorithm. Based on the 
original formulation Q21 of MKL as a QCQP, we propose a fast MKL algorithm (MwuMKL) that (a) does 
not require commercial solvers (b) does not make explicit calls to SVMs (unlike alternating optimization 
based methods), and (c) provably converges to an accurate solution in a fixed number of iterations. Our 
techniques draw on matrix multiplicative weight update (MWU) based approximation algorithms for SDPs. 
We make significant modifications to the algorithm to take advantage of the structure of the MKL problem; 
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these include a fast routine for exact matrix exponentiation, a way to do away with the binary search inherent 
in the MWU framework, and a number of optimizations to the primal-dual core of the MWU method. A 
detailed evaluation on eleven datasets shows that our proposed algorithm (a) has comparable accuracy to 
prior approaches (b) is significantly faster, especially as the data size increases beyond a few thousand points, 
and (c) compares favorably with the uniform heuristic that merely averages all kernels without searching for 
an optimal combination. As has been recently noted O, this heuristic is a strong baseline for the evaluation 
of MKL methods. 

Outline. We review the convex formulation of multiple kernel learning in Section[2] Section[3]describes 
both the MWU framework and our algorithm. In Section|4]we empirically evaluate the quality and scalability 
of MwuMKL. Finally, in Section|5]we present our observations and highlight future directions. 

Related Work. In practice, since the space of all kernels can be unwieldy, many methods operate by fixing 
a base set of kernels and determining an optimal (conic) combination. Pavlidis et al. iTTol proposed assigning 
equal preference to all kernels and simply using an unweighted sum of kernel functions (UNIFORM). In their 
seminal work (SdpMKL), Lanckriet et al. lTT2l proposed to simultaneously train an SVM as well as learn 
a convex combination of kernel functions. The key contribution was to frame the learning problem as an 
optimization over positive semi-definite kernel matrices which in turn reduces to quadratically constrained 
quadratic programming (QCQP). Soon after, Bach et al. H proposed a block-norm regularization method 
based on second order cone programming (SOCP). 

In a bid to improve efficiency, many MKL algorithms started using alternating optimization, alternating 
between updating the classifier parameters and the kernel weights. For example, Sonnenburg et al. [ 20 ] mod- 
eled the MKL objective as a cutting plane problem and solved for kernel weights using Semi-Infinite Linear 
Programming (SILP) techniques. Rakotomamonjy et al. [17] used sub-gradient-descent-based methods to 
solve the MKL problem (SimpleMKL). An improved level set based method (LevelMKL) that com- 
bines cutting plane models with projection to level sets was proposed by Xu et al. ll24l . More recently Xu 
et al. ll25ll derive a variant of the equivalence between group lasso and the MKL formulation that leads to 
closed-form updates for kernel weights (GroupMKL). 

Other works in MKL literature study the use of different kernel families, such as Gaussian families iPTBl , 
hyperkernels [ 14] and nonlinear families II22H 711. Kloft et al. ifTOlfTTTl and Vishwanathan et al. [23 ] introduce 
regularization based on the £ p norm. Cortes et al. [8 ] propose a two-stage approach to optimize the kernel 
weights in the first stage and learn a standard SVM in the second stage. In addition, Orabona and Luo lfl5l 
study online algorithms for MKL based on stochastic gradient descent. 

2 Background 

Let X = fx i,X2,...,x„) € R" xrf denote a collection of n training samples in a cf-dimensional space. Let 
y = {yi>y2, ■ ■ ■ ,y>n) £ {— 1,+1}" denote the binary class labels for the data points in X. The problem of 
multiple kernel learning can be cast as the problem of minimizing ^||w|| 2 such that yj({w,Q?(xj)) +b) > 1 
for all j where <I> is the feature map associated with the (unknown) kernel k. With the constraint that K can 
be written as fc = Y!iL\ (with a trace regularizer) the dual problem takes the following form lTT2l 
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minmax 2a 1 — a Ga 

K a 

m 

s.t. K = £ jU ; K,-, tr(K) = c, K y 0, /x > 

i=l 

where K, is the Gram matrix for X associated with the kernel function •), G, = diag(y)K, diag(y) and 
G = Y!i=\ Mi'Gj. The kernel K optimizing this expression can be found by solving the following convex 
optimization problem [12, Theorem 20]: 

max 2a T l -cs (2.1) 

s.t. s> — a T G,a, a T y = 0, a>0 

where r £ W n and tr(K,) = r,-. This program is an example of a class of convex programs called quadratically 
constrained quadratic programs (QCQPs). Lanckriet et al. [ 12] propose solving this program with a second 
order cone program (SOCP) solver such as Mosek [1] or SeDuMi [21J, as this type of solver is very efficient 
and the class of QCQPs is a subset of SOCPs. 

We note that ( |2.1[ ) is the hard-margin version of the MKL problem: there are soft-margin variants that 
are also amenable to our methods lTT2l . For the 1-norm soft margin, an additional constraint is added, namely 
that all the terms of a are bounded from above by the margin constant C. For the 2-norm soft margin, another 
term ^a T a appears in the objective, or we can simply add a constant multiple of the objective to each G ; . 

Notation. We will denote vectors by boldface lower case letters like z, and mauices by bold uppercase 
letters M. The operator y is used to denote positive semi-denniteness; M y states that M is positive 
semidefinite. The matrix inner product A • B = Tr(AB) = YtijAijBij- We denote a zero matrix (in boldface, 
like vectors and matrices), and an all-ones matrix 1. 



3 Algorithm 



The central idea of this paper is to solve the multiple kernel learning problem as formulated in ( |2.1| ) via 
the matrix multiplicative weight update method (MWU), specifically the approach for solving semidefinite 
programs (SDP) employed by Arora and Kale [3]. This method is a general template for solving such 
programs and requires a significant amount of customization and optimization for each specific instance. In 
what follows, we will describe how to apply it to the MKL problem, and highlight places where the structure 
of our problem allows us to optimize beyond what the original template allows. 



An SDP reformulation. The first step is to reformulate the QCQP in ( |2.1| ) as an SDP. While this is trivial 
in general (every QCQP is an SDP) we require a specific form of the SDP in order to apply the MWU. This 
form is given as 

min gz s.t. £ F J'Z/- H z > 0. (3.1) 

z j 

Let A^A, = ^G; for all i £ [0..m]. Since each G,- y and c > 0, all the A; exist. Algebraic manipulation 
of ( |2.1[ ) allows us to express it in the form of ( |3.1| >, with H = ^j" jj^ , z = (oc,s), Fj = .^') T ^0^1 
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and F„_|_i = f^ T ^ , where A ; (j) is the j th column of A,-. A more convenient formulation combines the 
matrix terms in the semidennite constraint into a set of matrix constraints Qj, yielding the form 

min cs — 2a T l (3.2) 



a.s 



I« A; Ot 



s.t. Qi(o) = ( (A ^ )T )h0 Vi € [l..m], a T y = 0, a > 0, 



Algorithm 1 MWU template Q 

Given primal variables LA 1 ) = (l/n)I, error e, # of iterations 7\ and guess ft) 
for ? = 1 . . . % do 
Run Oracle with L,W 
if Oracle fails then 

Return and increase ft) 
else 

zW <— dual variable obtained by ORACLE 
M® i- (ZjFjzf -H+pI)/2p 

w (m)^ (1 _ e) i; =1 MW 

T _ W''+') 

^ ~~ TrW('+ 1 > 

Return L T and decrease ft) 



The MWU framework. At a high level, the MWU is a primal-dual scheme that works as follows. It starts 
with a guess ft) for the optimal value of the SDR Assuming that this guess for the optimal value is correct, 
the program reduces to a feasibility problem in the primal and dual variables. The algorithm then attempts 
to find either a feasible primal or feasible dual assignment such that this guess is achieved. 

The search for feasible values is conducted by an exponential re-weighting search. The process starts 
with some (typically infeasible) primal solution. In each step, an oracle (which solves a simple linear 
program) is used to determine a candidate dual z. If this is possible, then the dual values are used to guide 
the search for a new (less infeasible) primal solution by exponentially re-weighting primal constraints that 
are more important, and the process repeats. 

If after a specific number of iterations the process completes, then we have found a solution for the 
current guess ft) and we can try reducing it. If at any step the process fails (because the oracle cannot find 
the desired z), then the guess was too low and is increased. This binary search over ft) eventually yields a 
solution guaranteed to be within a (1 + e) factor of the optimal solution. The inner routine (inside the binary 
search) is summarized in Algorithm [T](p is dependent upon ORACLE). 

In order to make the framework feasible and efficient for solving MKL, we need to solve a number of 
problems. In what follows we describe these one by one, and then present the final algorithm. 

3.1 Eliminating Binary Search 

The first optimization we can perform is to note that we can circumvent the binary search on ft) entirely and 
run Algorithm [T] only once. 
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Since every iteration of the binary search is reduced to a feasibility problem we can restate (2J_i as: 



co=(cs — 2a 1) s.t. j>-a T G,a, a y = 0, a > 0. 

r i 

We can further transform the problem by taking advantage of the KKT conditions. The support con- 
straints of the SVM problem can be written as Ga + by > 1. If we multiply both sides of this inequality 
by a T then it becomes an equality (by complementary slackness): a T Ga = a T l. cs is a substitution for 
a T Ga in the MKL problem lPT2"1 . so cs = a T l as well. The objective function is linear, so we can scale s 
and a with any positive factor that we like, and use the fact that cs = a T l = \(0\ to transform the problem: 

find a 

s.t. 1/(c|g)|) > — a T G ! a, & T y = 0, a T l = l, a > 0, 

T i 

where a = \co\a. The first constraint can be transformed back into an optimization; that is, 

min max — a T G, a , 

subject to the remaining linear constraints. Because co does not figure into the maximization, we can com- 
pute (0 simply by maximizing y& T Gj&. Practically, this means that we simply add the constraint a T l = 1, 
and the "guess" for CO is set to —1. We then know the objective, and only one iteration is needed, so the 
binary search is eliminated. 



3.2 Matrix Exponentiation 



A critical step in the MWU process is computing (1 — e)^'=i M ' (see Algorithm [lj). For MKL, is a 
block-diagonal matrix, with m blocks of the form My = ^(Q,-(a w ) + pl n +i). The matrix (1 - £)^=i M<f) 

will also be block-diagonal, with m blocks (1 — e)^*=i <' . Therefore we just need a way to exponentiate 
Mf = l p (Q ; -(I f T =1 oW ) + ptl n+l ) for each i. 
In general, matrix exponentiation is expensive, and Arora and Kale [3] suggest ways to approximate this 
computation. However, in our case, the particular form of Q, (c.f. ( |3.2[ )) yields a more elegant closed form 
solution: 

Lemma 3.1. The exponential of a matrix in the form ^\ , where a and b are nonnegative, is 

/(cosh sinh l/f cos y)uu T sinhi/Asin/u \ fl /l„ — uu T 0\ 
y sinhi/Asin/u 7 coshi/A — sinhi/^cosyy y 0y' 



where u is the unit vector u/||u||, <j) = (a + b)/2, iff = ^(a — b) 2 /4+ ||u|| 2 , and 7 = tan 1 (2||u||/(a — b)). 
Proof. We symbolically exponentiate an« + lx« + l matrix of the form 



M 



al u 
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Since this matrix is real and symmetric, its eigenvalues A; are positive and its unit eigenvectors v,- form an 
orthonormal basis. The method that we use to symbolically exponentiate it is to express it in the form 



M = £A ; v ; v 7 

!=1 



The exponential then becomes 



( <Svvf. 



1=1 



As a matter of notation, let u be the unit vector such that ||u||u = u. 

Eigenvalues. The characteristic equation for M is not difficult to calculate. It is: 

(A - a) n - 2 (X 2 - {a + b)X+ab - ||u|| 2 ). 



(3.4) 



This yields n — 2 eigenvalues equal to a, and the other two equal to (a + b)/2+ \/(a — b) 2 /4 + ||u|| 2 and 
(a + b)/2 — -\/ (a — b) 2 /4 + ||u|| 2 . We label them X\ and %i, respectively, and the rest are equal to a. 

Eigenvectors. To compute the eigenvectors of M, we compute the solution to (A,7 — M)v,. 
For Ai , we have 

\X X -a)I -u ^ (y\\ _ 



-u 7 X\ - bj \ 1 

(we will scale vi later). This yields v'j = u/ (X\ — a). We want to normalize this, so we compute 



+ 1 



|u|| 2 + (Ai - a) 2 



(Ai-a) 2 (Aj-a) 2 
(A 1 -a)(Ai-^) + (A 1 -a) 2 
(Ai-a) 2 

(Ai - b) + (Ai -a) _ 2Ai-Q + fr) 

X\ — a X\—a 
2X\ — (Ai +A2) X\ — A2 



Ai — a 



X\ — a ' 



(3.5) 



(3.6) 



where (|3.5|> results from X\ being a root of (|3.4|), and (|3.6[) is because a + ft = Ai + X%. So vi becomes 



X\ — a ' 



' Ai — a 
Ai — A? 



Ai — a 



VAi — aV Ai — A2 ' V ^1 — ^2 



\/Ai —ay/X\ —b „ I X\— a 
\/X\— aV Ai — A2 ' V Ai — A2 



' Ai-ft 
Ai — A2 



' X\—a 
X\ — Xi 
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The formula for v 2 is similar, except with X\ and A 2 reversed, and a sign reversal on v' 2 : 




(v' 2 ,l) / I h-b_ t I fo-a 



11(^,1)11 \ \X 2 -h 'VA2-A1 

We can quickly observe, however, that since A 2 — b = a — X\ and X2 — a = b — Xi, 

(v' 2 ,l) 



(v 2 ,i; 




We let a = J j^j^ and j8 = J (noting that a 2 + j8 2 = 1), so the unit eigenvectors become 

vi = (au,j3) v 2 = (-j8u,a) 

As for the other eigenvectors, we only care about L" =3 v,-vf , since all other eigenvalues are equal. But 
this is just 

n n 

ET T T T t T T 

V i V i =L V ' V ' _V 1 V 1 ~ V 2V 2 =4+l-VlV 1 -V 2 V 2 

i=3 i=\ 

_ (a 2 uu T aj3u\ _ / j8 2 uu r -a/3u 
~ 1,1+1 V«j3u r P 2 ) \-ocpn T a 2 

(M T 0\_//„-M T 
-4+1 lJ"V r 

The Exponential. All that remains is to put everything together: 

e M = £e\ i vf 

(=i 

_ v / a 2 uu r aj8u\ ^ / j8 2 uu r -aj8u\ a //„ - uu r 
~ e \apu T j3 2 J +e \-apu T a 2 J + \ T 

Some variable substitutions will give us the form in (|3.3|); X\ = + A 2 = — y/, and a = cos(y/2): 



/(coshi//^sinhi/Acosy)uu T sinhi//sinyu \ „//„ — uu r 0\ 
y sinhi/Asinyu r cosh l/f — sinhy/ cos yj \ 0/ 



□ 



In each iteration, the matrix to be exponentiated is a sum of matrices of the form 5^ (Q* (1^=1 05 ) + 



p?I„ + i), and so Lemma 3.1 can be applied. While the derivation of the exponential is straightforward, 



practical consideration needs to be given to our implementation of the exponential. 



In Lemma 3. 1 note that if we give large inputs to the functions exp, cosh, and sinh we will rapidly 
overflow even a double-precision range. Fortunately exp(x)/2 gets exponentially close to both sinh(x) and 
cosh(x), so above a certain value, we can use exp alone and throw in a "quashing" factor on the 

result, because it will be normalized out later in the computation. 

We can also simplify the exponentiation by observing that \(o\ = 1. This makes = «m = and 
V = 0, which then makes Xjfi proportional to ||u,||, cos(y) = 0, and sin(y) = ±1 (in our case this is —1 ). 
Algorithm[2]contains the pseudocode for the exponentiation. 
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Algorithm 2 EXPONENTIATE-M 



Require: y, a, {G,}, e', p, t 

0<"-2ji(l + P)* 
for i G [l..m] do 
||u ; || <- \fa7Gia 

& <~ M\ GiCC 

q <— max,- 
ifq< 20 then 
for i G [l.jn] do 
// 1 «- cosh(v^) 

// 2 < sinh(w) 

e M <- 1 
else 

for i G [l.jn] do 

'i ^ l i 

eM <— 2e~ 9 {This is effectively 0} 
5^m(n-l)e M + 2y: ; -/, 11 
for i G [l..m] do 

f, n <-//7* 

Z/ 2 <- Z, 12 /S 

g<-L-2Z/ 2 g,- 

return lj , g 



3.3 Computing The Oracle 

We now need an implementation for the ORACLE routine. For an SDP given by ( |3.1| ) with current primal 
"guess" L, the oracle returns a z > such that EyC^/ •L)z/ > H»L. Translating this to the context of MKL, 
we need an oracle that given L expressed as a block diagonal matrix with blocks L,, finds a such thafj] 
L-Gi(a) # L,->0,a > 0,a T y = 0,a T l = 1. 



Note that L comes from the process of exponentiation, and so has special structure that Lemma 3.1 
allows us to exploit. Recall that u, = A,a and li, = u,-/||uj||. Let us set 

lj 1 = ^(coshi/A + sinh vcosy), l} 1 = e^(sinhyAsiny), if 2 = ^(cosh — sinhy/cosy) 

With algebraic manipulation, and the observations that Uj&J has unit trace and that a, is the same for all i, 
£; (Fj • ~L)zj > H • L becomes: 

(m \ m 

£(2//^) a > -m(n - l)e a -^{l} 1 +lfcs). (3.7) 
i=0 / i=0 



Since cs = \G)\ = 1 from Section 3.1 the right-hand side of this inequality is simply the negative of the trace 



1 The MKL formulation introduces additional linear constraints on a that are not strictly part of the original SDP. It is easy to 
show however that these can be folded into the oracle computation. 
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of L, so this simplifies to 

(£(2Z, 12 uTA ; -) JaW>-l, (3.8) 



\i=l / 

since L is normalized with trace 1 in the MWU template. 



Algorithm 3 Oracle-MKL 
Require: y, g 

Ensure: a > 0, a T l = 1, a T y = 

P<^{i\ji = l},N<^{i\ji = -l} 
i P i- argmax^pg,-, i N <- argmax^g, 
a <— 

a ip <- 1/2, a iA , <- 1/2 

if a satisfies inequality (^ ; - 2Z/ 2 g ; ) T « > — 1 then 

return a 
else 

return FAIL 



Satisfying Equation ( |3.8| ) We have reduced the oracle problem to finding an a that satisfies Equation ( |3.8| ) 
in addition to the other (linear) constraints on a. Further simplifications make this equation even more 
tractable. Firstly, observe that u^A,- can be rewritten as gj = (c/r,) 1 / 2 a T G,/(a T G ! a) 1 / 2 , allowing us to 
simplify Q to (I,-2// 2 g,) T a > -1. 

Note that the coefficient vector g = gj can be easily computed when we exponentiate the matrix 

M^. In order to find a a that satisfies (3^1, we simply choose the highest elements of g that correspond to 
both positive and negative labels, then set each corresponding entry in a to 1/2. If this choice of a does not 
satisfy ( |3.8[ ), then we know that the oracle must fail. 

Algorithm [3] describes the pseudo-code of ORACLE for MKL. 



Notes. The oracle procedure produces a very sparse update to a: in each iteration, only two coordinates 
of a are updated. This means that each iteration is very efficient, taking only linear time. Further, the trick 
of replacing u, by g ( means that we never need to explicitly compute Af, which in turn means that we do not 
need to compute the (expensive) square root of Gj explicitly. Another beneficial feature of the ORACLE for 
MKL is that terms involving the primal variables L are either normalized or eliminated, which means that 
we never have to explicitly maintain L. 



3.4 Extracting the solution from the MWU 

To extract the kernel weights, we start by ob serving that £"Li Q 
Performing similar algebra as in Section 



further to 



3.3 



£a T G,-a(2£ ( 12 ) 



we see that 2£™ l f ^a T G,a 

(c/tf N 1/2 



0, by complementary slackness. 

1 . We transform this 



?12 



!=1 



a T G,-ce 



1. 
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This form suggests that 2i) 2 ( y' 1 ' ) is the appropriate choice for Hi. Indeed, since ELj jU,-a T G,-a is 
a T Ga, and because of the KKT conditions of the original problem, we know that a T Ga = a T l = |w| = l. 

3.5 Putting it all together 



Algorithm 4 MWU-MKL 

Require: gW = 0; 

p, the width of ORACLE; 

e, the desired approximation error 

Set e' = -ln(l-^) 

SetT= ^ln(n) 
repeat {t times} 

Set a {t) = ORACLE(y,gW) 

Update a = a + aW 

if Oracle failed then 
return 

SetMf ) = 2 L(Q,-(aW)+pI„+i) 

Set W, W = e - e '%=i M ? ] {Call ExpONENTiATE-M(y, a, {G,-}, e', p, 0, Algorithm^ 

SetLf +1) =wf ) /Tr(wf ) ) 

Compute g(' +1 ) from {G,}, and a 

until t = t 
return \a,~L^ t+ ^ 



Algorithm [4] summarizes the discussion in this section. The parameter e is the error in approximating 
the objective function, but its connection to classification accuracy is loose. We set the actual value of e via 
cross-validation (see Section [4]). The parameter p is the width of ORACLE, a parameter that indicates how 
much the solution can vary at each step, p is equal to the maximum absolute value of the eigenvalues of 
Q ( -(a (f) ),for any/Q. 

Running time. Starting innermost, every iteration of Algorithm [4] will require a call to ORACLE-MKL, 
a call to Exponentiate-M and an update to G ( a and a T G,a. Oracle requires a linear search for two 
maxima in g, so the first is 0{n). The latter are each 0(mn), which dominate ORACLE. Algorithm|4]requires 
a total of T iterations at most, where T = ln(«). The parameter p 2 can be bounded as 0(c): 

Lemma 3.2. p is bounded by 1 + y/c/2. 

Proof, p is defined as the maximum of [|Q(ff^)[| for all t. Here || • || denotes the largest eigenvalue in 
absolute value [3]. Because cs = \co\ = 1, the eigenvalues of Qj(cu^) are 1 (with multiplicity n — 1), and 
1 ± ||A,-aW || . The greater of these in absolute value is clearly 1 + [|A,-aW ||. ||A,gjM || [ s equal to 
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OjW always has two nonzero elements, and they are equal to 1/2. They also correspond to values of y 
with opposite signs, so if j and k are the coordinates in question, (a^) r G i -aW < (l/4)(G,( ;i ) + G,-( J y.))> 
because G,-(,-n and G,vjy) are both negative. Because of the factor of c/r,-, and because r, is the trace of G„ 
||A £ -ccW [| < y/c/2. This is true for any of the i, so the maximum eigenvalue of Q(gjW) in absolute value is 
bounded by 1 + s/c/2. □ 

Since we only require one run of the main algorithm, the running time is bounded by O [cmn\n(n)\^ . 



4 Experiments 

In this section we compare the empirical performance of MwuMKL with other multiple kernel learning 
algorithms. Our results have two components: (a) qualitative results that compares test accuracies on small 
scale datasets, and (b) scalability results that compares training time on larger datasets. 

We compare MwuMKL with the following baselines: (a) UNIFORM (un-weighted combination of 
kernels), (b) SdpMKL HI, (c) SimpleMKL El, (d) LevelMKL El, (e) GroupMKL [25]. Another 
related method is SMOMKL ll23l : we discuss this in Section|5] We evaluate these MKL methods on binary 
datasets from UCI data repository. They include: (a) small datasets Iono, Cancer, Pima, Sonar, Heart, Vote, 
WDBC, WPBC, (b) medium dataset Mushroom, and (c) comparatively larger datasets Adult and CodRna. 

Classification accuracy and kernel scalability results have been presented on small datasets (with many 
kernels). Scalability results (with 12 kernels due to memory constraints) have been provided for medium 
and large datasets. Due to space limitations, only a subset of our results on small datasets are presented. In 
addition, for ease of readability, we compare a smaller set of MKL methods (UNIFORM, GroupMKL and 
MwuMKL) in the main part of the paper. These MKL methods are the fastest among all baselines, as can 
be seen in the detailed comparisons presented in the appendices for all baselines on all datasets. For data 



scalability results, we first compare all baselines (see Figure [12] of Appendix |A|) on moderate size dataset 
Mushroom. For larger datasets Adult and CodRna, we compare MwuMKL only with UNIFORM which is 
the fastest among all baselines (as we show for Mushroom). In all cases, we omit the results for SILP ll20l . 
which takes significantly longer to complete (e.g. on Sonar, the smallest dataset in our pool, each iteration 
of SILP takes about 4500 seconds on average, whereas UNIFORM requires 0.03 seconds on average). 

Similar to lfT71l25ll . we test our algorithms on a base kernel family of 3 polynomial kernels (of degree 1 
to 3) and 9 gaussian kernels (of width {2~ 3 ,2- 2 , . . . ,2 5 }0). For small datasets, we use kernels constructed 
from individual features (i.e., a kernel is constructed by considering only one feature in the computation, 
setting all other features to zero) as well as from all features taken together. For medium and large datasets, 
due to memory constraints, we test only on 12 kernels constructed using all features. All kernels are nor- 
malized to trace m (the total number of kernels). For all experiments we report results averaged over 30 
iterations (with standard deviations where applicable). In each iteration, 80% of the examples are randomly 
selected as the training data and the remaining 20% are used as test data. Feature values of all datasets 
have been scaled to [0, 1]. SVM regularization parameter C is 100 and is chosen by cross-validation. The 
duality gap (required as a stopping criterion for many baselines) as well as other settings are same as de- 
fined in the SimpleMKL toolbox U3. For MwuMKL, we set e = 0.2. In theory, MwuMKL requires 
T = (8p 2 /£ 2 )ln(«) number of iterations. However, we observed that in practice MwuMKL converges in 
much smaller number of iterations and we have found t/32 iterations to be sufficient for good performance 
on all of our datasets. Both e and the iteration factor were chosen by cross-validation. Contrary to existing 



technically since we are normalizing the range of each feature, we should use a smaller bandwidth set for each kernel; in future 
work we will release results with a bandwidth range of {2~ 8 ,2~ 7 , . . . ,2°}. 
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works we do not compare the number of SVM calls (as MwuMKL does not explicitly use an underlying 
SVM) and the number of kernels selected. 

Experiments were performed on a cluster of 30 machines with two different configurations: (a) 10 
quad-core Intel Xeon E5405 CPUs with 32GB of RAM, and (b) 20 single-core Intel Xeon CPUs with 8GB 
of RAM. All methods have an outer test harness written in MATLAB. The methods we compare against 
are implemented using libSVM [5] based SVM solvers and MOSEK (ht tp : //www . mosek . com)"] based 
convex solvers (we note that these solvers are all implemented and optimized in C). MwuMKL also uses a 
test harness in MATLAB with an inner core written in C. 
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(a) Cancer with m = 120 kernels 
Figure 1: Cancer (n = 683, d ■ 



(b) Pima with m = 108 kernels 
: 9) and Pima (n = 768, d = 8). 



Accuracy. On small datasets our 
goal is to show that MwuMKL com- 
pares favorably with other baselines 
in terms of test accuracies. How- 
ever, the asymptotically better train- 
ing times of MwuMKL is not ap- 
parent due to the small sizes of these 
datasets. Hence, we demonstrate 
the superior scalability of MwuMKL 
on larger datasets. We present 
test accuracy results for UNIFORM, 
GroupMKL and MwuMKL on the 



two largest among the small datasets, namely, Cancer (Figure[Ta|) and Pima (Figure [Tb]). In each case, we re- 
port the number of kernels used. For both datasets, MwuMKL yields good accuracies. Additional detailed 
results (Figures 4b lib in Appendix |A|) show that the accuracy of MwuMKL is either the best or close to 
it. The computational benefits for MwuMKL is not noticeable for Cancer which has a small training time. 
For Pima MwuMKL is significantly faster than GroupMKL and quite close to UNIFORM. 



Data Scalability. We start with results for Mushroom (n = 8124, d = 112) with 12 kernels. Mushroom 
is linearly separable and as expected all methods in Figure [2a] achieve 100% accuracy. However both 



MwuMKL and UNIFORM are much faster as compared with GroupMKL. Detailed results in Figure 12 
(of Appendix [A]) show that MwuMKL and UNIFORM are the fastest among all baselines and are signif- 
icantly faster. Hence, for the remaining experiments on large datasets, we only compare UNIFORM with 
MwuMKL. 
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(a) Mushroom with m = 12 kernels 
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Figure 2: Results for Mushroom (n = 8124, d = 112), Adult (n = 9768, d = 123) and CodRna (n = 9525, 
d = S). 
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Figures 2b and 2c present results for Adult (n = 48842, d = 123) and CodRna (n = 59535, d = 8), 
respectively. Note that although these datasets are large, with 12 kernels (on all features), we could only 
experiment on a subset of the entire data (9768 points for Adult and 9525 points for CodRna) due to memory 
constraints. On both datasets, MwuMKL is much faster when compared to UNIFORM and the trace lines 
indicate that the gap (between training times of UNIFORM and MwuMKL) will only get wider with further 
increase in data size. 

For Adult, the accuracy of MwuMKL is comparable to Uniform. For CodRna, the accuracy of 
MwuMKL is better than UNIFORM for smaller data sizes but is slightly worse (though within error bounds) 
for larger data sizes. 
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Figure 3: Time and Accuracy ver- 
sus #kernels for Pima. 



Kernel Scalability. We compared the scalability of MwuMKL 
with increase in number of kernels on small datasets. In Figure [3] 
we present kernel scalability results for Pima (the largest among the 
small size datasets). As can be seen, MwuMKL is among the fastest 
while yielding the best accuracy. Kernel scalability results on addi- 



tional datasets have in provided in Appendix |B| (Figures [T3p0| ). Ex- 
cluding Heart, in all other cases, the accuracy of MwuMKL is either 
the best or among the best. We also observed that changing C from 100 
to 10 makes MwuMKL achieve the best accuracies for Heart. The 
time taken by MwuMKL on small datasets is comparable to UNI- 
FORM and GroupMKL. We believe that the benefits of MwuMKL 
(in terms of running time) with large number of kernels would be per- 



ceivable on bigger datasets (as is true for Pima in Figure [3] and Figure 15 ) 



5 Discussions 

Our results show that MwuMKL, while closely matching the accuracy of prior methods, runs significantly 
faster than these approaches, and is even faster than the simple UNIFORM heuristic for larger datasets. We 
believe that this substantially advances the state-of-the-art for fast MKL methods. Note that, with precom- 
puted kernels, MwuMKL allows us to get to the largest datasets possible before memory limitations show 
up: this is clearly the next bottleneck in the quest for MKL methods that truly scale. One idea that we 
expect to be helpful is to make use of dynamic kernels, or even redesign the MWU scheme to explicitly only 
deal with rows of the kernel matrices instead of the entire matrix. In this regard, SMOMKL [23] presents 
interesting results on the large datasets we used, although they present no accuracy numbers, and their stated 
running times appear to be significantly larger (we were unable to complete a detailed comparison using our 
computational platform). 
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A Accuracy Results on additional datasets 
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Figure 4: lono (n = 351, d = 33) with m = 12(33 + 1) = 408 kernels. 
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Figure 6: Pima (n = 768, d = 8) with m = 12(8 + 1) = 108 kernels. 




ii 



120 
_100 
§ 80 



^Uniform 
SDP 
Simple 

-0- Level 

—Group 

^Mwu 




Number of points 

(a) Time versus data size 



100 



80 



o 60 

CD 
Z3 

o 

2 40 



20 




50 100 150 
Number of points 

(b) Accuracy versus data size 



200 



Figure 8: Heart (n = 270, d = 13) with m = 12(13 + 1) = 168 kernels. 
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Figure 9: Vote (n = 435, d = 16) 
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(b) Accuracy versus data size 
with m = 12(16 + 1) = 204 kernels. 
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Figure 10: WDBC (n = 569, d = 30) with m = 12(30+1) = 372 kernels. 
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Figure 11: WPBC (n = 198, d = 33) with m = 12(33 + 1) = 408 kernels. 
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Figure 12: Mushroom (n = 8124, d = 112) with m = 12 kernels. 



B Kernel Scalability Results on additional datasets 
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Figure 13: Iono (n = 351, J = 33) with m = 12(33 + 1) = 408 kernels. 
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Figure 15: Pima (n = 76S,d = 8) with m = 12(8 + 1) = 108 kernels. 
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Figure 16: Sonar (n = 20S,d = 60) with m = 12(60+ 1) = 732 kernels. 
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(b) Accuracy versus number of kernels 
Figure 17: Heart (n = 270, d = 13) with m = 12(13 + 1) = 168 kernels. 
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Figure 20: WPBC (n = 198, d = 33) with m 
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